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1. 

This  paper  strives  to  provide  a  sound  underpinning  for  the  theory 
of  stochastic  games.  Section  2  is  a  reworking  of  the  Bevley-Kohlberg 
result  Integrated  with  Shapley's;  the  "black  magic"  of  Tarski's  principle 
is  replaced  by  the  "gray  magic"  of  the  Hilbert  Nulls tellensatz.  Section 
3  explicates  the  underlying  topology  and  measure  theory;  I  believe  it  is 
as  necessary  for  Mertens  and  Neyman's  proof  of  the  minmax  theorem  [8]  as 
it  is  for  mine  [9,  10].  Finally,  Section  4  establishes  a  result  on  this 
sort  of  structure  which  may  be  of  some  independent  interest;  in  any  case, 
it  is  critical  for  the  argument  in  {10]. 

This  work  was  strongly  influenced  by  Andrew  Gleason.  Other  acknow¬ 
ledgments  may  be  found  in  [10]. 
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2. 

Fix  a  stochastic  game.  In  this  section  we  study  the  rediscounts 
game,  following  [11]  and  [3]. 

Let  S  ,  the  state  set,  A  (resp.  B  )  the  choice  set  each  turn 

for  A  (resp.  B  ),  and  C  ,  with  fixed  measurable  function  q  :  [0,1]  -*•  C 

all  be  finite  sets;  let  4  :  S*A*BxC-*-S  and  d:S*A*BxC-*-  [— fif,  ft] 

give  the  state  and  the  outcome  resulting  from  choices  (starting  in  state 

s  £  S  )  of  a  £  A  ,  b  £  B  and  random  c  6  C  .  Without  loss  of  generality, 

these  sets  and  functions  describe  the  game  completely.  Consider  now  the 

stochastic  game,  with  fixed  starting  state  s  ,  and  discount  rate 

00 

r  €  (0,1)  ;  that  is,  let  the  payoff  function  be  ][  d.  (l-r)*~  »  where 

th  iml 

dA  the  payoff  on  the  i  step. 

As  £  d  (1-r)  is  a  continuous  payoff  function  on  a  pair  of 
i-1  1 

compact  strategy  spaces  (see  Section  3)  the  min-max  theorem  follows  immed¬ 
iately  (this  well-known  result  follows  rapidly  from  [11]);  let  the  value  of  the 
game,  for  starting  state  s,  be  Vg(r)  .  Following  [3]  and  [11]  we  characterize 
the  values  and  optimal  strategies,  by  considering  the  system  of  equations: 

U  (r)  *  val(Exp(d(s,a,b,c))  + (1-r)  y  P(s) »U-(r))  .  (2.1s) 

8  C  seS  8 

Here  P(s)  *  measure({t  6  [0,1]  ;  4(s,a,b,q(t))  -  s})  depends  upon  a  , 

b  ,  and  s  ;  val  is  just  the  ordinary  min-max  value  of  the  matrix. 

For  fixed  r  ,  think  of  the  system  as  one  equation  in  a  variable  in 
|S| 

c'  1  .  Applying  the  Contraction  Mapping  Theorem  Q6],  p.  229),  it 
must  have  a  unique  solution  (l?s(r))  .  Indeed,  as  the  contraction  con¬ 
stant  ||  1— r  ||  is  bounded  away  from  1  on  any  interval  [r*,  1]  ,  and 
Indeed  on  a  set  (x  +  yi  €C  •  r*^x^l,  0^y_<  d(r*)}  ,  (&s(r))  is 
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a  continuous  function  of  r  in  (a  neighborhood  of)  (0,1)  . 

Proposition  2.2:  For  r  &  (0,1)  ,  this  solution  is  precisely  (Vfi(r))  . 

Proof:  It  is  clearly  enough  to  show  the  following,  for  fixed  r  €.  (0,1)  : 

By  playing  as  his  i**1  move  an  optimal  strategy  in 

the  game  given  by  the  matrix  of  (2.1si_1),  Mr.  A  (2.3) 

guarantees  himself  a  payoff  of  at  least  U  (r)  . 

80 


(2.3)  is  clearly  equivalent  to: 


ft  (r)  >  V  (r)  for  all  s-  £  S  . 
80  “  80  0 


(2.4) 


So  suppose  (2.4)  is  false.  Let  <*  »  max(V  (r)-u  (r))  ;  choose  s 

S  80  80 

so  that  the  max  is  achieved. 

Then 


iMr)  -  val(Exp(d(s,a,b,c))  +  _J  (l-r)P(s)  *?f-(r)) 
C  s€S  8 


-  -(l-r)o  +  val(Exp(d(s,a,b,c))  +7  (l-r)P(s)  * (1^— (r)  +a)) 
C  ieS 


>,  -(l-r)o  +  val(Exp(d(s,a,b,c) )  +  (l-r)P(s)  *V-(r))  . 

C  seS 


So 


^(r)  +  a  >  val(Exp(d(s,a,b,c))  +  _][  (l-r)P(s:a,b,s)  *V-(r))  . 

C  seS  8 

(2.5) 


But  since  both  sides  of  (2.5)  equal  Vg(r) 


we  have  a  contradiction. 

□ 


It  is  well-known  that  the  value  of  a  matrix  is  one  of  a  finite  list 
of  rational  functions  of  the  coefficients  (namely  the  value  of  some  square 
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submatrix;  see  18],  p.  76).  Thus  the  system  may  be  viewed  as  a  set  of 

polynomial  equations — Identifying  S  as  il,  ... ,k)  ,  where  k  »  )s |  — 

®l^xl*  ***,xk^  *  °»  •••»  B^Cx^,  “  0  »  where  the  coefficients 

lie  in  the  algebraically  closed  field  F  .  Now,  if  the  ideal 

k 

(gj^*  •••»  g^)  — that  is,  {  £  h^  :  hjC  F  ,  i  ■  1 . k)  — equals  the 

1-1  k 

unit  ideal  (1),  then  some  linear  combination  £  h.g.  -  1  ,  h,  £  F 

4b1 

1  1  k  k 

for  i  »  1,  ...,  k  ;  this  is  impossible  because  £  h.g, ((V  ))  «  7  h .  *0  =  0 

1*1  1  1  6  i-i  1 


when  viewed  as  functions  of  r  . 

By  the  Hilbert  Nullstellenstaz  (weak  form;  see  [2],  p.  69),  then, 
the  only  other  possibility  is  that  ((x^,  . . . ,  x^)  6  F^  :  g^(x^,  . . . ,  x^) 

*  0,  ...»  g^(x^,  ...»  x^)  ■  0}  is  non-empty.  On  the  other  hand,  we  saw 
above  that  the  solution  is  unique  in  the  much  larger  field  of  all  functions 
of  r  ;  hence  it  is  certainly  unique  in  F  ,  and  precisely  equal  to 
(Vfl(r))  .  But  an  algebraic  function  which  maps  reals  to  reals  must  in¬ 
deed  lie  in  F’  *  the  field  of  real  algebraic  functions  C  F  .  Finally, 
observe  that  each  V  (r)  equals,  at  each  point,  one  of  a  finite  list  of 
algebraic  functions;  each  pair  of  these  can  cross  each  other  only  finitely 
often  (for  their  difference  is  an  algebraic  function,  which  can  only  have 
finitely  many  zeroes).  Thus  by  the  continuity  of  the  Vg(r)  ,  there 
exists  an  Interval  (0,r)  on  which  the  Vg(r)  are  algebraic,  for  all 
s  C  S  . 


F*  has  a  natural  ordering:  f  >  g  if  and  only  if  there  exists 

%  k 

an  r  such  that  f  >  g  on  the  Interval  (0,r)  .  Consider  the  games 
(2.1s)  with  the  values  Vg(r)  substituted  in  place  of  the  (Jg(r)  ;  then 
these  are  ordinary  two-person  zero-sum  games,  with  coefficients  in  F*  , 
and  hence  have  optimal  strategies  {f(a,s;r)>  ;  by  construction,  for 
each  a  and  s  ,  f(a,s;r)  belongs  to  F'  .  For  each  r  ,  then,  we 
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have  optimal  stationary  strategies,  algebraic  in  r  ,  for  the  r-discount 
game.  Finally,  it  is  well  known  that  algebraic  functions  have  Pulseaux 
(fractional  power)  series  expansions— see  [1 ] — and  so  we  have  checked 
the 

Theorem  of  Bewley  and  Kohlberg:  The  values  of  the  r-discount  game,  and 
the  stationary  optimal  strategies,  have  Pulseaux  expansions. 
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3. 

More  generally,  consider  an  Infinite  decision  tree  such  that,  at 
each  node,  the  choice  set  is  finite.  By  possibly  duplicating  certain 
branches,  we  nay  assume  that,  without  loss  of  generality,  there  exist 
finite  sets  ,  i  -  1,  2,  ...»  such  that  Z^  consists  simply  of  the 
first  node,  Z ^  is  the  choice  set  at  the  first  node,  Z^  is  the  choice 
set  at  the  second  node,  no  matter  which  node  was  chosen  as  the  second 
node,  and  so  on.  Let  Q  be  the  space  of  paths  on  the  tree; 

Q  *  zi*^2x***  is  compact  by  Tychonoff's  theorem  (f9],  p.  144). 

Simultaneously  fix  probability  distributions  on  the  choice  set 
at  each  node.  Then  these  distributions  induce  a  sequence  of  functions 
^i  :  Z1x...xZ1^*  such  that 


z£Z 


ui<2l* 


6i~r 


*> 


pi-l^2l’ 


2i-l) 


for  any  fixed  z^,  ...,  ,  for  any  i  ■  1,  2 . 

The  following  is  a  special  case  of  the  Kolmogorov  Extension 
Theorem  CI5J). 


Proposition  3.1:  There  exists  a  measure  p  on  Q  such  that 
p  ( { ^  { ^2 }  x  ...  x  {z^}  x  x  Z^+2  x...)  "  Pi^l*  * '  *  *  9  for  all 

z^,  . . . ,  ,  for  all  1  . 

Proof:  Consider  the  ring  generated  by  Z  -  the  collection  of  sets  of 
the  form 


{*1>x...x{z1}xZ1xZ1+1x...  . 


We  clearly  have  a  finitely  additive  set  function  p  with  the  desired 
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properties;  to  check  that  y  is  a  measure,  we  need  only  confirm  that 
for  any  descending  tower  D  YjD  ...  ,  Yj  £  Z  for  all  j  , 

m 

n  Y  «  $  implies  lim  y(Y.)  *0  ([4],  p.  39).  But  every  element  of 

j-1  2  J-*-  J 

2  is  both  open  and  closed  in  the  direct  product  topology;  hence  the  same 
is  true  of  every  member  of  the  ring.  Thus,  by  the  compactness  of  Q  , 
any  tower  descending  to  0  has  Y^  -  $  for  all  j  sufficiently  large; 
hence  lim  y(Y.)  *  0  trivially.  Finally,  since  Z  is  a  base  for  the 

j-HK>  J 

direct  product  topology,  the  Caratheodory  Extension  Theorem  extends 
to  the  Borel  field  ([4],  p.  54). 

□ 

Denote  by  t  a  function  which  assigns  to  every  node  in  N  ,  the 
set  of  nodes,  a  probability  distribution  on  its  choice  set;  denote  the  set 
of  all  possible  t  by  U  ,  the  space  of  mixed  strategies.  Then  every 
t  €  M  induces  a  probability  measure  y^  on  Q  .  Let  P  £  M  ,  the  space 
of  pure  strategies,  be  the  set  of  all  it  e  M  such  that,  at  every  node, 
it  selects  some  alternative  with  probability  one.  Then  there  exists  a 
canonical  mapping  f  :  P  -*•  Q  ,  sending  neP  into  the  path  determined 
by  x  .  x  also  induces  a  probability  measure  ©T  on  P  ,  and  we  have 

eT  -  wT?f  • 

Partition  N  into  two  disjoint  subsets  U  Nc  where  will 

be  thought  of  as  the  set  of  decision  nodes,  and  as  the  set  of  chance 

nodes.  Then  we  can  write  M  ■  Mg  x ,  where  an  element  tb(tc)  of 

is  an  assignfemnt  to  each  node  in  Wfi(Wc)  a  probability  distribu¬ 
tion  on  its  choice  set.  Similarly  P  ■  P„  . 

6,  .  then  decomposes  as  a  product  measure: 

'tB*  ~C' 
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0 ,  .  -  0  x  e 

(  B'V  TB  1 


Assume  now  that  tc  is  fixed.  Letting  dA  ,  i  -  1,  2,  ...  be 
a  sequence  of  measurable  functions  on  Q  ,  bounded  below,  we  have 


Proposition  3.2: 


inf  lim  inf  / 


TBeMB 


,  N 

wi-l  1 


dp 


A'V 


inf  lim  inf  / 

V?B  Q 


i  N 

ll-i 

Ni»l  1 


dp 


Thus  thinking  of  as  the  nodes  controlled  by  the  decision-maker  and 

Nc  as  those  controlled  by  nature  (with  nature's  "strategy"  known  in  ad¬ 
vance,  the  decision-maker  "might  as  well"  play  ^  pure  strategy  as  a  mixed 
one.  This  is  true  even  in  a  slightly  stronger  sense: 

Corollary  3,3:  If  there  exists  a  mixed  strategy  such  that  the  infinum 
is  achieved,  then  it  is  also  achieved  by  a  pure  strategy. 


Proof  of  Proposition  3.2:  For  any  i  ,  for  any  Tg  , 

/  did'J(x  r  )  m  f  di(f(ir))d0(T  _  )<»> 

Q  1  UB,TC'  P  1  UB'TC' 

-  /  (/  d.  (f  (tt_  ,  n_))d0  (nr))d0  (ir_) 

r.  Pc1  s  c  T»  c  tb  b 


by  Fubini's  Theorem  ([91,  p.  233).  Thus  of  course 


1  N 

i  A 


i«l 


dp 


^TB’TC* 


-  /  /. 


P„  P„ 
B  C 


1  N 

£  l  di 

Ni-1  1 


d  0  d0 
TC  tb 
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Observing  that  the  support  of  6w  is  a  subset  of  P_  on  which 
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positive  integer.  So  the  set  vhere 

f,  N 

lim  inf  /  ±  l  d  d6  ft  I 

N-*»  pclNi“l  J  C 

is  indeed  of  measure  zero.  Since  the  support  of  6  is  of  course  of 

B 

measure  one,  we  are  done. 

□ 

Note  3.4:  The  results  of  this  section  remain  valid  if  "finite"  is  replaced 
by  "compact." 

Note  3,5:  Our  "mixed  strategies"  may  not  be  the  most  general  the  decision¬ 
maker  could  adopt.  However,  strategies  with  more  complicated  forms  of 
mixing  can  be  disposed  of  in  the  same  fashion  using  the  compact  form  of 
Proposition  3.2  (see  Note  3.4). 
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4. 

Let  {Z^}  ,  Q  ,  {y^}  ,  and  y  be  as  in  the  previous  section. 
Suppose  each  Z^  has  a  distinguished  subset  Z*  such  that 
(Z^  x  ...  x  Z  x  Z*  x  (Z*+^)C  x  z*+2  x  • • •  )  ■  0  ;  In  other  words,  once  a 
node  in  some  Z*  is  reached,  the  path  remains  in  the  {Z*}  with  prob¬ 
ability  1.  For  a  given  path  z  -  (z^  z2,  ...)  ,  let 


P*(i;z) 


* 


x  {2i-i}  xZi  xZ 


i+1 


y({z  }  x 


,Ul-l,xZi”Zl+x“ 


if  z,  ,  4  Z 


i-1  ^  i-1 


if 


the  probability  (conditional  on  z^,  ...,  z^_^  )  of  entering  Z*  .  Suppose 
we  wish  to  integrate  a  function  over  Q  which  vanishes  outside 
(Z*)c  x  (z*)c x  . . .  .  Then,  intuitively,  we  can  integrate  a  smaller  func¬ 
tion,  on  a  new  measure  which  is  bigger  on  (Z*)c  x  (z^)0  x  ...  ,  to  achieve 
the  same  result.  This  can  be  helpful  if  we  wish  to  estimate  the  integral 
by  estimating  the  (smaller)  function  directly. 

More  precisely,  we  prove  this  result  in  terms  of  truncated  sequences; 
it  is  applied  in  Section  5  of  JlQJ.  For  any  z  -  (z^,  ...  )  £  0  , 

define  the  truncations  z(k)  «  (z^,  ...,  z^)  £  Z^  x  ...  x  z^  .  Write  also 
Q(z^»  ...»  2^)  —  {z^}  x  {Z2}  x  ...  x  x  x  Z^+2  x  •  •  •  C  Q  •  Then 

f\j 

Proposition  4.1:  There  exists  a  probability  measure  y  on  Q  such  that, 
for  all  k  ,  for  all  f^  :  Z 
f^  5  Z^  x  ...  x  Zj^j^  x  Zk  {0} , 

k 

/  f.  (z(k))dy  -  /  fk(z(k))-  n  (1  - P*(i;z))dy  . 

Q  K  Q  i-1 


^  x  . . .  x  z^  •+■  R  such  that 
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Proof:  By  Proposition  2.1(  to  specify  v  we  need  only  specify 

uCQCz^,  *t))  ,  for  all  z^,  ...»  ,  for  all  1  ,  while  checking 

that 

_  E  •••»  2^i>  z) )  ■  P(Q(tp  ...,  2^^))  (4*2) 

Z1€21 


So  we  define  p(Q(z^,  ....  z^))  inductively  on  k 
P(Q)  -  1  .  Fix  z  -  (z2,  ...)  6  Q 


p(Q(z1)) 


nr(ir.rp(<l<*i))  lf  2i*zI 


if  z1  e  zj 


(4.3) 


unless  P^(l;z)  *  1  .  In  that  case. 


p(Q(z1))  -  P(Q(Zj))  • 


More  generally,  suppose  we  have  specified  y(Q(z^,  . ..,z^  ^))  .  If  this 
equals  0,  we  let  y(Q(z^,  ...,  z^_^,  z^)  ■  0  .  If  not,  then  also 
v(Q(z1,  ....  t  0  »  and  we  let 


p(Q(z^,  •••»  2^) ) 


j  p(Q(z1, 


*  zi_lJ) 


l-P*(i;z)'p(Q(z1,  ....  zi_1))*y(Q(zl’ 

if  x±  t  Z* 


if  xi  €  zj 


9  z . )  ) 


unless,  again,  PA(i;z)  ■  1  ,  in  which  case  we  let  y(Q(z^,  ...,  z^)) 
■  vWfzj,  ....  z±))  . 


Now 
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_  I  m(z^»  •••*  zj_i»  zj^ 


Zl€Zi 


^  v(Q(z^,  •••»  z^_^) )  ^  _ 

1  -  P*(i;z)  ,w(Q(*1 . z^)) . Zi-1»  2l})  * 


But 


*ik 


y(Q(z^,  •  ••  i  z^) )  —  (1 ~  P^(i j z) ) y (Q(z^,  •  •  •  i  ) 


by  the  definition  of  PA(l;z)  ,  and  so  (4.2)  is  checked,  and  we  have 

'V. 

defined  a  measure  y  . 

It  remains  to  show  that 


/  f,  (z(k))dy  -  /  f.  (z(k))*  n  (1  -  P*(i;z))dy  . 
Q  Q  i-1 


We  start  by  observing  that 


J  "  X  / 

Q  zi*‘«-xzk  Q(z1*...xzk ) 

k 

But  on  Q(zn  x  ...  xz  )  ,  f  is  just  a  constant,  as  is  n  (l-PA(i;z))  , 

i-1 

and  hence  we  are  asking  whether  £f(z^,  •••»  z^) •y(Q(z1,  ....  zfe)) 
k  ^ 

-  ^(z^,  ....  z^)*  n  (1  -  P^djzJJuCQCZj^,  ...,  zfc))  .  This  of  course  follows 
if 


y(Q(z^,  ....  z^))  -  Jl  (l-P*(i;z))*p(z1,  ....  Zj^))  (4.4) 


whenever  ffc(z^,  ....  z^)  +  0  — i.e.,  whenever  z^  ^  z£  . 

When  k  •  1  ,  (4.4)  is  immediate  by  (4.3).  So  assume  proved  for 

k-1  ,  and  check  for  k  .  Then,  for  any  z^  $  Z*  ,  we  define  y  on 
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